ebooktestfandomcom-20200214-history
Introduction to Developing Raptor (SBSv2)
This page contains information that is useful for anyone wishing to develop Raptor, the build system for Symbian OS. "We" usually refers to the main Raptor development team from Nokia. Technologies Raptor is written in Python and Gnu Make (with supporting Bash shell scripting) with XML configuration files. Also, there is a small number of supporting tools written in C (Talon and the Descrambler - see below). Developers should have a good working knowledge of C, C++, Python and Make, and some knowledge of Bash shell scripting and XML. The Python code is compatible with Python 2.5 and 2.6. We expect to move to using Python 3.x with Raptor at some point in the future, so the general advice is to use the newest Python available to you. At the time of writing, Python 2.6.4 is recommended. Raptor's make files are only compatible with Gnu Make 3.81, and they rely on features only supported in this version of Gnu Make. This version is the latest one and has been around for a while, so obtaining it should be relatively simple. The Raptor configuration files are all in XML. Suggested development environment We recommend using the following development environment: * Eclipse with CDT mainly for the syntax-highlighted makefile editor * PyDev - a Python IDE plug-in for Eclipse While not absolutely essential, the PyDev/Eclipse combination has some very nice features, and the PyDev Python debugger is excellent. Essential dependencies for development are * Python - preferably Python 2.6.4. * Gnu Make 3.81 (Windows: Use MinGW, Linux: part of build-essentials on .deb-based distros, or its own package on .rpm-based distros) * Gnu Compiler Collection C/C++ (Windows: Use MinGW's version, Linux: part of build-essentials on .deb-based distros, or its own package on .rpm-based distros). * Core utils/GNU Tools (Windows: Use Cygwin but be sure to not use Cygwin's make, Linux: standard GNU tools) * Appropriate compilers to build for the target platform(s) (e.g. ARM RVCT or GCCE) ** GCCE may be obtained from CodeSourcery's website http://www.codesourcery.com/sgpp/lite/arm/portal/subscription3058 * Mercurial (Windows: Download from the Mercurial website; Linux: install from your package manager). You can also use TortoiseHg which is a cross-platform GUI front-end to Mercurial and provides good shell integration. Binary versions of Python may also be obtained from ActiveState. Some people like to use Vim for editing instead of Eclipse/PyDev, and it's possible to show correct syntax highlighting in that too. XML, Make and Python syntax highlighting is usually available and turned on. For convenience, you can associate files with the .flm and .mk extensions with the same syntax-highlighting rules as Makefiles. This is a straightforward configuration option in Eclipse, and for Vim a simple addition to your .vimrc file enables this. For Windows, Vim is available for download from the link above; for Linux, install it from your package manager if required - many distributions have it installed by default anyhow. Note All Raptor's Python modules use tabs for indentation rather than space. You should set up your IDE/text editor to use tabs. Version numbering scheme Raptor version numbers are generally of the form 2.X.Y, with X and Y being integers. If a release only contains bug fixes since the previous release, the "micro" version number, Y, is incremented by 1. If new features are included in a release, the "minor" version number X is incremented and the "micro" version number Y is set to 0. The following sequence illustrates this for the imaginary releases 2.15 and 2.16: 2.15.0 New features, and possibly some bug fixes 2.15.1 Bug fixes since 2.15.0 2.15.2 Bug fixes since 2.15.1 2.15.3 Bug fixes since 2.15.2 2.16.0 New features, and possibly some bug fixes Occasionally, a special beta-release is made for testing purposes and its version number (and version string) might contain some other text to indicate this. Obtaining the source code The Raptor source code is kept in a Mercurial repository and is part of the Build Package. The URL is http://developer.symbian.org/oss/FCL/sftools/dev/build/ You can simply clone this Mercurial repository using hg clone http://developer.symbian.org/oss/FCL/sftools/dev/build --pull This will create a 'build' folder in your current directory, so move to the directory where you want this placed before running the command. Repository layout and the branches of Raptor TODO - add comments about submitting changes/testing? Raptor is found in sbsv2/raptor inside the repository. It is in the same place regardless of which branch of the repository you are viewing. The build repository contains three branches: * default * 'fix' - for defect fixes * 'wip' - feature work in progress Our normal development practice is to make changes in the fix and wip branches as appropriate, and merge these into the default branch after doing sufficient release testing. It's possible to create your own branches but given Mercurial will tend to create a new sub-branch whenever a new set of changes is made, this is rarely necessary. Setting up the development environment Please see our How to get and build Raptor from source page. How Raptor Works Raptor consists of essentially four main parts: Shell script/batch file command-line interface, Python Code, FLMs (Function-Like Makefiles) and XML Configuration files. The following diagram gives an overview of the interactions between the different parts of Raptor. The aim of this section is to provide an architectural overview of how each of the parts of Raptor work, not only in isolation, but together as the build system. The sbs shell script and sbs.bat batch file As already indicated, the sbs shell script (Linux) and the sbs.bat batch file (Windows) provide the command line interface to Raptor. The sbs shell script is also used on Windows when Raptor is used in a Cygwin shell, usually Bash. For brevity, the term "shell script" will be applied to both the sbs shell script and the sbs.bat batch file, and "shell scripts" will be used to refer to both of these. Broadly, the purpose of the shell scripts is to set the correct environment for the build. The main environment settings required are HOSTPLATFORM and HOSTPLATFORM_DIR. These determine the locations of various host utilities. For Windows, this means Cygwin, MinGW and Python. For Linux, this means the bundled versions of Bash, Gnu Make, Python, Talon, PVM and PVMGMake. On Linux, it should be noted that the supporting shell script $SBS_HOME/bin/gethost.sh is used to determine the values of HOSTPLATFORM and HOSTPLATFORM_DIR. Once all the environment variables have been set, the $SBS_HOME/python/raptor_start.py Python script is launched with the arguments given. At the time of writing, there is some commonality between the sbs shell script and sbs.bat batch file and it is likely that it will be merged into the sbs shell script with the sbs.bat batch file invoking the sbs shell script via Cygwin's Bash. Setting initialisation files TODO Python Code The Python code in $SBS_HOME/python is referred to as the "front end" of Raptor. Its main roles are * Metadata parsing: parse build information files (bld.inf's) and MMP files, including preprocessing etc * Tool checking: check that specified tools are available and recent enough to work well with Raptor * Perform exports: copies out all header files from bld.inf's. * Writing out Raptor's makefiles * Writing out XML log files * Passing log files to filter modules to provide customisable output The main entry point to Raptor is the Python module $SBS_HOME/python/raptor_start.py. This is a very simple wrapper that imports the "raptor" module ($SBS_HOME/python/raptor.py) and calls the Main function from that module. $SBS_HOME/python/raptor_start.py also enables developers to use the Python profiling module, cPython. To enable profiling, create an environment variable called SBS_PROFILE_BASENAME which should be the path to an output file for the profiling data. It is assumed that all directories exist, and the profile data file will be created as needed. For example, on Linux you might do this: export SBS_PROFILE_BASENAME=~/profile_data_raptor_build.out and on Windows something like set SBS_PROFILE_BASENAME=%USERPROFILE%\Desktop\profile_data_raptor_build.out $SBS_HOME/python/raptor_cli.py handles parsing of the command line options. The Raptor module $SBS_HOME/python/raptor.py does all the setting up for the build: it parses the sbs_init.xml files if they exists, sets defaults, and acts on the command line options passed to it from $SBS_HOME/python/raptor_cli.py. Plug-in Filters The output from Raptor comes from a number of sources such as the Python interpreter, the make process, child processes of make etc. Raptor's architecture is carefully constructed to ensure that output is valid XML, and the full XML output is always present. However, the full XML output is not always needed. For example, for the terminal output the full XML is far too verbose. Thus, Raptor includes a plug-in filter feature. As the name suggests, these are dynamically loaded at runtime (hence "plug-in") and their purpose is to sieve (hence "filter") the XML output. The filters are Python classes that implement a certain interface. Specifically, this interface is defined by the following class: class Filter(object): def open(self, params): return False def write(self, text): return False def summary(self): return False def close(self): return False def formatError(self, message): return "sbs: error: " + message + "\n" def formatWarning(self, message): return "sbs: warning: " + message + "\n" which is defined in $SBS_HOME/python/filter_interface.py. Any class that implements this interface it referred to as a filter. You may have as many filters as you wish in any Python module, as long as their names are unique. By default, Raptor uses two filters: filterterminal and filterlogfile. These are defined in the Python modules $SBS_HOME/python/plugins/filter_terminal.py and $SBS_HOME/python/plugins/filter_logfile.py. As you might expect, these filters are for terminal output and log file output. You may use any number of filters at a time specifying them with the --filters option - this option is case-insensitive so it will find FilterTerminal even if you specify filterterminal. The mechanism works by passing output as it arrives in the Python interpreter's buffer to each filter and letting it do something with that output. For the log file filter, this is to simply write the data to the log file. In the case of terminal filter, the output is filtered and much is thrown away to produce informative build output like this example from > sbs -b simple/bld.inf -c armv5 compile : simple/test1.c++ armv5_urel compile : simple/test.cpp armv5_urel compile : simple/test2.cxx armv5_urel compile : simple/test3.Cpp armv5_urel compile : simple/test4.cc armv5_urel compile : simple/test5.CC armv5_urel compile : simple/test6.C++ armv5_urel compile : simple/test.cpp armv5_udeb compile : simple/test1.c++ armv5_udeb compile : simple/test2.cxx armv5_udeb compile : simple/test3.Cpp armv5_udeb compile : simple/test4.cc armv5_udeb compile : simple/test5.CC armv5_udeb compile : simple/test6.C++ armv5_udeb target : epoc32/release/armv5/urel/test.exe armv5_urel target : epoc32/release/armv5/udeb/test.exe armv5_udeb no warnings or errors Run time 20 seconds sbs: build log in /home/user/sdk-01/epoc32/build/Makefile.2009-11-30-13-26-41.log whereas the log file /home/user/sdk-01/epoc32/build/Makefile.2009-11-30-13-26-41.log is 54,260KB in size. Note: the filters are called synchronously so they need to be efficient and return quickly. Currently, filters are located in $SBS_HOME/python/plugins, but a mechanism will be added soon to allow them to be used from the SDK being built against, probably from a subdirectory of epoc32/sbs_config. Make As already mentioned, Raptor uses Gnu Make as its build engine. In fact, it's possible to use other make engines that are compatible with Gnu Make 3.81. By default, Raptor generates one main makefile with name Makefile_all in $EPOCROOT/epoc32/build. This is usually a very short makefile that simply includes several others. Here is an example from a Windows build: # GENERATED MAKEFILE : DO NOT EDIT MAKEFILE_GROUP:=DEFAULT # GROUPER MAKEFILE ALL:: include Z:/epoc32/build/Makefile_all.export include Z:/epoc32/build/Makefile_all.bitmap include Z:/epoc32/build/Makefile_all.resource_deps include Z:/epoc32/build/Makefile_all.resource include Z:/epoc32/build/Makefile_all.default # END OF GENERATED MAKEFILE : DO NOT EDIT Separation into included makefiles like this allows Raptor to easily define the various targets such as BITMAP, RESOURCE, EXPORT etc. Each of these included makefiles contains the main machinery for doing builds: Function-Like Makefiles. Function-Like Makefiles (FLMs) As their name implies, function-like makefiles are used like functions: a list of variables is set for that makefile and then that FLM is included. This is similar to many other programming languages, where functions define their arguments and these are set before the function is run. When Make parses the Makefile_all, as it parses the included makefiles it evaluates all the variables and then expands them for the included the FLM. To see this, here is a cut-down extract of Makefile_all.default from a Windows build: # call C:/raptor/lib/flm/e32abiv2exe.flm SBS_SPECIFICATION:=simple.mmp SBS_CONFIGURATION:=armv5_urel COMPONENT_META:=C:/raptor/test/smoke_suite/test_resources/simple/bld.inf COMPONENT_NAME:= COMPONENT_LAYER:= PROJECT_META:=C:/raptor/test/smoke_suite/test_resources/simple/simple.mmp DATE:=C:/raptor/win32/cygwin/bin/date.exe DUMPBCINFO:= GNUMAKE38:=C:/raptor/win32/mingw/bin/make.exe GNUCP:=C:/raptor/win32/cygwin/bin/cp.exe GNUCAT:=C:/raptor/win32/cygwin/bin/cat.exe GNUMKDIR:=C:/raptor/win32/cygwin/bin/mkdir.exe GNUMV:=C:/raptor/win32/cygwin/bin/mv.exe GNURM:=C:/raptor/win32/cygwin/bin/rm.exe GNULN:=C:/raptor/win32/cygwin/bin/ln.exe . . . MODULE:=simple USER_LIBS_PATH_OPTION:=--userlibpath VARIANTPLATFORM:=armv5 PLATFORM:=$(VARIANTPLATFORM) VARIANTTYPE:=urel VERSION:=10.0 VERSIONHEX:=000a0000 VFE_OPTION:=--no_vfe EXPLICITVERSION:= TARGETTYPE:=exe UID1:=1000007a include C:/raptor/lib/flm/e32abiv2exe.flm MAKEFILE_LIST:= # work around potential gnu make stack overflow In this case, the Raptor FLM in question is C:/raptor/lib/flm/e32abiv2exe.flm. In this build, 224 variables were set for this FLM. Internally, the FLM consists of a number of Make macros that are evaluated in the context of these variables. These produce the Make rules for a particular executable that is built from the MMP file C:/raptor/test/smoke_suite/test_resources/simple/simple.mmp. The make macros expand to rules to build the specified target. These are written in Bash shell script. For example, here is an extract from $SBS_HOME/lib/flm/e32abiv2.flm that deal with generating the ARTARGET: ## Link-type selection: # runtime static libraries link via AR ifneq ($(ARTARGET),) # Assuming that there are no libdeps in this case because this is probably one of the # Runtime libraries which has no deps. define artarget_func $(ARTARGET): $(if $(MULTIFILE_ENABLED),$(MULTIFILEOBJECT),$(LINKOBJECTS)) $(STDCPPTAGFILE) $(if $(MULTIFILE_ENABLED),,@echo "$(STDCPPTAGFILE)" > $(VIAFILE); $(call groupin10,$(LINKOBJECTS)) ;) $(call startrule,ar,FORCESUCCESS) \ $$(call dblquote,$(AR)) $(ARCHIVER_CREATE_OPTION) $$@ $(if $(MULTIFILE_ENABLED),$(MULTIFILEOBJECT),$(COMMANDFILE_OPTION)$(VIAFILE)) \ $(if $(DUMPBCINFO),&& $(FROMELF) -v $$@ > $$@.elfdump,) \ $(call endrule,ar) endef $(eval $(artarget_func)) CLEANTARGETS:=$(CLEANTARGETS) $(VIAFILE) $(if $(DUMPBCINFO),$(ARTARGET).elfdump,) endif The 224 variables set for this FLM are defined by the XML interfaces discussed below. XML configuration Raptor has two main types of XML configuration files: those that contain "variants", "groups" and "aliases", and those containing "interfaces". These are defined in various XML files that are located in $SBS_HOME. The "variants", "groups" and "aliases" are defined in the XML files stored in $SBS_HOME/lib/config, whereas the "interfaces" are alongside their associated FLMs in $SBS_HOME/lib/flm. Note that Raptor also reads XML files from $EPOCROOT/epoc32/sbs_config so that it is possible for some "variants", "groups", "aliases" and "interfaces" to be defined there. XML configuration - Variants, Groups and Aliases "Variants", "groups" and "aliases" are, very roughly, the command line values passed to the -c option and tell Raptor which configurations to build for. For example, sbs -c armv5 -c armv5.test builds armv5 production and armv5 test code. These dot-separated strings such as armv5 and test can be either a "group" combined with zero or more "variants", or an "alias" combined with zero or more "variants". "Variants", "groups" and "aliases" are defined as XML tags in Raptor's XML files in $SBS_HOME/lib/config. For example, there is a "v5" variant element that defines characteristics of the AMRv5 architecture (see $SBS_HOME/lib/config/arm.xml): Here are two aliases: Aliases provides shortened and simplified names for other complicated chains of variants. For example, we alias armv5_urel to the verbose arm.v5.urel.rvct2_2. The meaning attribute of an alias provides the "expansion" of that alias so wherever armv5_urel is used, Raptor will know that is really means arm.v5.urel.rvct2_2. Groups provide a way to collect several convenient build configurations together. For example, the following XML tag defines the group called default: Using this group as the option to -c will build armv5 and winscw together. XML configuration - Interfaces TODO - I think this should live closer to the FLM documentation, but the document needs a little refactoring for that to happen. "Interfaces" are a bit like function declarations in C/C++: they define the list of parameters that an FLM takes, i.e. which variables must be set for that FLM to obtain a meaningful result. If any variables in an interface are undefined, Raptor throws an error. Interfaces are defined in XML "interface" tags. For example, the Symbian.lib interface defined in the following extract of the file $SBS_HOME/lib/flm/standard.xml: Notice that the interface element has has two attributes: extends and flm. The extends attribute gives the name of another interface, in this case Symbian.e32abiv2. This, as you would expect, indicates that Symbian.lib "extends" or inherits all the input parameters (param tags) from the interface Symbian.e32abiv2, and defines three extra ones of its own. The key parameter defined here is the TARGETTYPE: this is matched to the value of the MMP keyword TARGETTYPE (by the metadata parsing) and tells Raptor which interface to use for a given MMP. There is one interface for each MMP target type. As implied, the interface elements form a tree via the extends attributes, each one gaining the parameters of the one it extends and possibly adding some of its own. The Symbian.pdl and Symbian.plugin interfaces (see $SBS_HOME/lib/flm/standard.xml) are examples of interfaces that extend from others but do not require any further parameters. The flm attribute tells Raptor which FLM to use for the given target type, and together with the parameters, generates the call to the FLM. Most of the variables in the interface get their values from the metadata (bld.inf and MMP files) and once Raptor has parsed all of these, it can write out the FLM call into a makefile. As already noted, the above interface extends the Symbian.e32abiv2 interface (see $SBS_HOME/lib/flm/standard.xml). This in turn extends yet another interface: Symbian.mmp. Following the extends tree upwards, we reach . . . The base.flm interface is the "root" or "base" interface that other interfaces inherit from. The abstract attribute indicates that the interface cannot be used directly and should always be extended by another interface. Notice the name attribute has value "base.flm" - this is often a cause of confusion as it is suggestive of a file name. There is no file named base.flm; this is just a name. This interface defined in the file $SBS_HOME/lib/flm/base.xml. The discussion so far has been focussed on the ARM target. There is an equivalent tree of interfaces for the WINSCW target defined in $SBS_HOME/lib/flm/emulator.xml and the discussion applies equally well to those. Supporting tools and host utilities TODO Talon Talon is a shell wrapper sitting between make (GNU make or emake) and the bash shell where build commands are run. As an illustration, a shell command issued by make to execute the compiler and build a particular file will be passed to the bash shell through Talon. Talon can then control bash output, enclosing it in descriptive XML tags and holding it until no other proceess are writing to the log. Talon can also timeout on long commands and retry failed commands. Features such as retry on timeout add reliability to GNU make. Talon superceded "sbs_descramble" as the way of serialising output. Talon is designed in such a way as to use less memory than the prior descrambler based mechanism in Raptor. Environment Variables Talon is controlled by environment variables but many parameters must be supplied that are specific to a particular recipe, such as the recipe name or the name of the MMP it's from. Talon needs these values to print correct xml tags. Some variables can be set for the whole build while others are specific to each recipe. There is a mechanism to pass recipe-specific parameters via the commandline that make gives to Talon. Talon strips these parameters off and then passes the remainder of the command to bash. Variables that are set for the whole build: * TALON_BUILDID - a unique number generated by the build system that identifies this build. Used for naming the semaphore that controls access to the log. * TALON_SHELL - what shell to execute recipes in * TALON_RECIPEATTRIBUTES - specifies what attributes to set in the recipe output and what environment variables to use in those attributes. e.g. could be "name='$RECIPENAME' component='$COMPONENT_META'" * TALON_DESCRAMBLE - if set to '1' then obtain a semaphore before showing output from the recipe. prevents recipe output from being interleaved in the log. Variables that may be set per-recipe * TALON_FLAGS - a space-separated list of words or "flags". Valid flags be "rawoutput" to allow descrambling without any XML. Also "forcesuccess" to ignore the success or failure of a command. * TALON_RETRIES - How many times to retry a failed recipe * TALON_TIMEOUT - how many milliseconds to wait before giving up on a build command. * TALON_DEBUG - if set, this causes talon to print out debugging information. Note that this generally messes up a build system so it's only useful in small tests. How Recipe Specific Parameters are Passed in Each command needing xml wrapping and descrambling must start with the following section: "|name1=value1;name2=value2;...;| " For Example make runs the following: talon -c "|name=compile;COMPONENT_META=fred/group/bld.inf;PROJECT_META=barney. mmp;| armcc -o betty.o betty.cpp" The startrule/endrule FLM macros take care of adding the recipe-specific-variables in "||". Talon strips off the bit between pipe symbols and uses that to know what the recipe name is etc etc. If talon doesn't see the "||" section then that means that make is executing a "$(shell)" instruction in the makefile - it goes quiet and tries to transparently pass the commands through. It does buffer the output from the bash command but it doesn't use a semaphore to check when to output it. No XML is produced. The Descrambler TODO Binary Variation C-Pre-Processor TODO Category:Raptor Build System